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Abstract 



In this research repon the effectivity of prototypes of interactive learning environments (ile) 
are compared with printed learning environments or 'traditional' face-to-face lectures. The 
learning content is the same for all conditions: a course on inferential statistics for first year 
universitary students. The computer-based and the printed environments are used for 
independent learning. In the learning materials, represented in the prototypes, a clear 
distinction is made between the basic content and embedded support devices (ESD) that are 
expected to support learning. The prototj^pes differ in relation to the extent they support 
interactivity in manipulating the ESD and the degree of discernability of the ESD. In a large 
empirical research set-up ILE are compared with learning environments that are based on 
printed learning materials and a control situation with face-to-face lectures. The results 
indicate that there is no difference in learning outcomes when the conditions are compared. 
They also show the role of the ESD and the significant impact of individual differences 
between students on study outcome. 
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1 Introduction 

This research report is based on the resulu of a collaborative study between the Open 
university of the Netherlands (Ou) and the University of Gent, Belgium. This reports 
presents a comprehensive part of this research. Among other aims, the study conducted 
focuses on the evaluation of interactive learning environments (ILE) to support independent 
learning. For an extensive overview of the study we refer to Martens, Portier, Valcke, Van 
Buuren, Dekeyser, Goeminne & Schuyten, 1995, a, b). 

1.1 Background 

In higher education there is growing need for more (cost)effective educational 
techniques such as independent learning. Often budgets for eduction are cut while drop-out 
rates among students are too high. This is certainly the case for 'difficult' subjects such as 
statistics. In an attempt to make education more effective and to be able to handle larger 
numbers of students, a research was carried out to investigate if independent learning is a 
good alternative for traditional face-to-face lectures. These lectures cost relatively much time 
from professors and from students (for instance because of travelling time). There is a 
growing pressure or institutes for higher education to increase the quality and decrease costs 
per student (van Driel & van den Berg, 1994; Segers, 1993). By researching techniques for 
making higher education more effective, this study aims and contributing to this issue. 

In this research report we concentrate on design characteristics of ILE and printed 
materials in relation to student use of support in ILE and student characteristics. As will be 
briefly discussed in the theoretical base of this study, student control over the learning 
environment is considered to depend on specific student characteristics (cf. e.g., Esichaikul, 
Smith & Madey, 1994; De Diana & Van der Heiden, 1994). Design features have to take this 
into account. 

To measure the effectivity of the ILE, three prototype-versions are used in a research 
set-up. Moreover, two alternative learning environments based on printed learning materials 
(PLE, printed learning environments) and a control setting with lectures are incorporated into 
the research design. The learning environments are researched with university students 
taking a statistics course. This course focused on 'inferential statistics' which is a part of an 
Ou-course for the social sciences. 

1.2 Learning materials to support independent learning 

The Ou sets up education in a distance education setting. Thus far, the delivery of education 
is mainly based on printed learning materials. The Ou has invested a lot in enhancing the 
quality of the learning materialrto 'embed' all kinds of support to help students during their 
study. The didactical elaboration, embedded in the learning materials is supposed to take 
over the supportive role of a teacher who is normally present during a lecture or a working 
group. 

Considering the independent learning setting, the high-quality elaboration of the 
learning materials is of prime importance to support a learning process: 

We distinguish in the learning materials on the one hand 'basic content' that reflects 
the domain specific (scientific) information and on the other hand embedded support 
devices (ESD). Given the large amount of ESD in learning materials, these are clustered 
according to their basic functions and effects as suggested by Valcke, Martens, Poelmans 
and Daal (1993): 

- orienting ESD: learning objectives, references to other learning materials, references to 
required prior knowledge, and use of history. 

- procc"ing ESD: additional learning mprerials, advance organizers, figures, glossary, 
introductions, study advices, summaries, tables, examples, extended learning 
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materials. 

- testing ESD: self-test items, exercise items (on know, insight and apply-level), answers. 
ESD are integrated into the basic content where course material developers consider 
them to be of relevance. Taking account of our theoretical base (see below) and earlier 
empirical studies (e.g., Portier & Van Buuren, in press), the level of integration into the 
basic content is considered to be an important variable. Therefore, we distinguish 
between learning materials in which ESD are hardly discernable and learning materials 
where the ESD are explicitly discernable or identifiable. Discernability of support 
devices may have an important impact on the way students use the material because 
with discernable ESD students can decide in advance to use or to skip ESD, whereas in 
the other mode the ESD has to be read. But discernability might also internipt the 
visual coherence of the study material and hinder the study process. 
In printed learning materials, ESD are available depending on the design-decisions of the 
learning material developers. In interactive (electronic) learning environments, we 
transfer the decision and responsibility to incorporate ESD into the basic content to the 
student. In our theoretical base, this means that the degree of 'interactivity' is 
considered as a basic feature of the learning environment. In the research set-up, this is 
reflected into sets of experimental conditions where two basic delivery modes are 
distinguished: printed delivery and interactive delivery. 

The explicit distinction between basic content and ESD, the level of ESD-discernability and 
the degree of interaction in manipulating ESD form the main objectives of the present 
research. Previous research has shown that use of ESD leads to better study results. But can 
they 'replace' a teacher? With this research we hope to gain more insight in this matter. 



2 Theoretical background of the study 

Our theoretical base positions learning with ILE in a framework of interrelated processes and 
variables. The instructional design model of Snow and Swanson (1992) is helpful to describe 
this theoretical base. These authors define the following interactions (represented by '"■') 
between variables and processes: 

uptitudes learning types content domain instructional design '^situations ^^populations 
Applying this matrix to our research, we get the following: 

aptitudes: cognitive and non-cognitive (motivation) aptitudes 

learning type: independent learning in interactive or printed learning environments 

content domain: statistics in social sciences 

instructional 

design: learning materials where ESD are separated from the basic content 

situation: luJopendent learning 

population: adult students 

Depicting this matrix results in the following theoretical model to describe and explain independe 
learning based on ILE (Figure 1): 
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Figure 1 A model for independent learning. 

2.1 Elements of the model 
This model comprises: 

- COGNITIVE APTITUDE related to LEARNING as described by the tliree-component theory of 
Sternberg (1986). This cognitive psychological theory is helpful to describe cognitive processes related to 
learning.-In earlier studies we have described how this theory can be related to the functions and effects 
of ESD in learning materials (Valcke, Martens, Poelmans, & Daal, 1993). 

- COGNITIVE APTITUDES related to READING. Since the textual representation of the learning 
materials is still a predominant representation mode, even in ILE, reading comprehension is an important 
process. The focus is on discourse comprehension and not on the decoding processes. In our model, 
reading comprehension is considered as a basic process related to the learning process, but also as an 
individual variable that interacts with the other variables and processes in the model. 

-The influence of the NON-COGNITIVE APTITUDE motivation. For example Ames and Ames (1984) 
have indicated the overall importance of motivation in relation to learning. Also in the context of 
independent learning and the specific design of learning materials, authors refer to the central role of 
motivation. Waller (1979) points out at motivational effects of typographic access structures for 
educational texts. Elshout-Mohr (1991), Van Hout-Wolters and Willems (1991) mention that motivation 
can be enhanced by presenting a specific set of ESD, such as objectives, questions/tasks, examples, 
introductions, feedback, content pages and structure pages. 

Considering the specific task environment, also the 'attitude towards learning with computers' is a 
relevant variable. 

3 
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- The LEARNING MATERIALS that are used as input for the learning process (basic content + ESD). In 
the research literature, a vast set of theoretical and empirical studies help to base the functions and effects 
of embedding support devices in learning materials. Also at the Ou various research projects and studies 
have been set up (Valcke, et al., 1993; Martens, Poelmans, Daal, Valcke & Stolk, 1994; Martens, 
Poelmans, Daal, Van Staa & Van Meurs, 1993; Martens, Poelmans, Daal, & Valcke, 1993; 1994). In 
general it was found that ESD in Pi,E are highly used, appreciated positively and lead to better study 
results. 

- The TASK-ENVIRONMENT: the interactive learning environment. An interactive learning 
environment is defined as a context that supports learners to interact with a knowledge base in order to 
attain clearly defined learning objectives. Support is especially realised by the possibility to adapt to the 
individual learner. 

ILE as conceptualized in our research are related to a variety of instructional technology applications. 
There is for instance a strong relationship with Hypercourseware' when the focus is on learner control 
and the availability of a large knowledge base (cf. Mayes, Kibby & Anderson, 1'590; Thompson, 
Simonson & Hargrave, 1992). ILE have analogies with 'microworlds' and 'ILE' as defined by Lawler (1987) 
when we concentrate on the active exploration and manipulation of a rich knowledge base. Also, ILE 
have a strong connection with (Intelligent) Computer Assisted Instruction, Adaptive Learning 
Environments Qones, 1992) and (Intelligent) Tutoring Systems (McCalla, 1992) when the focus is on the 
adaptive qualities (Larkin & Chabay, 1992). 

ILE resemble Computer Managed Learning systems and especially the emerging Interactive On-Line 
Advisory systems when we consider the 'adaptive' possibilities of the system. 

- INDIVIDUAL VARIABLES that interact with study behaviour and the way the learning environment 
is used and processed. In the present research report we focus on a subset of individual variables: reading 
comprehension, prior knowledge and use of ESD. The variable reading comprehension has already been 
discussed. The second variable refers to the prior knowledge a student already has before starting the 
learning process. Many authors have indicated the importance of this variable (e.g., Ausubel, 1968; Mayer, 
1979; Dochy, 1992; Dochy & Alexander, 1995). Dochy (1992) reveals in this context the multi- 
dimensional nature of prior knowledge. In our research we will for instance make use of the distinction 
between the behavioural and knowledge dimension in prior knowledge. The former refers to the 
'mastery' level a student can perform in relation to a specific body of knowledge (knowing, insight in, 
application). The latter refers to the specific 'type' of knowledge he or she masters (facts, concepts, 
relations, structure, method). The use of ESD can be measured by means of computer log-files, that 
indicate what ESD are used. 

2.2 Research hypotheses 

There are many possibilities of adapting ILE to learner characteristics. To ensure the comprehensibility of 
this report only a subset of possible interactions will be reported on. In this research subjects are not 
assigned to certain conditions based on individual variables, '■^ut are ascribed randomly to research 
conditions. 

Considering the theoretical base and the specific design of our ILE and PLE, the following hypotheses are 
put forward. The first three hypotheses focus on the effectivity of the learning environments used. The 
last three hypotheses focus on the interaction between student characteristics, the use of the learning 
environments and study outcome. 

1 Subjects who study in ILE will achieve at least equal study outcomes as subjects who study in PLE 
or face-to-face lectures. 

2 Subjects who use ESD benefit from learning environments where ESD are less discernable. In contrast, 
subjects who do not use much ESD benefit from learning environments where ESD are discernable. 
This interaction is expected because low-users may want to skip certain ESD. When ESD are non- 
discernable this might hinder these learners because they have to read the ESD before they know 



The term 'Hypercourscware' is used to envelop software products which provide some combination of hypertext capabilities, 
e.g., HyperCard^", SuperCard^", LinkWay™, Guide^", ToolBook™ and Plus^". 
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what ESD it is. On the other hand high-users who always tend to read ESD might find the different 
lay-out of ESD obstructive. 

Students who study in ILE achieve higher study outcomes than students using a non-interactive (but 
still electronic) learning environment. 

This expectancy is based on the idea that the action of choosing ESD leads to higher awareness of 
functions ot these ESD. 

Prior knowledge is positively related to the use of functionalities of the ILE and is a significant 
interaction variable in the relation between treatment and study outcome. 

Previous research has sho'.vn the important impact of the variable prior knowledge on the learning 
process. We expect subjects with high prior knowledge to explore more in ILE, and therefore to 
make more use of ESD in ILE. We also expect a significant interaction with prior knowledge in the 
relation between characteristics of the learning material and learning outcome. 
Good readers use less ESD to support their study activities; reading comprehension is an interaction 
variable in the relation between treatment and study outcome. No predictions are made about the 
direction of the interaction. 

Motivation and ESD-use are interrelated positively; their interaction influences study outcome. No 
predictions are made about the direction of the interaction. 
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3 Research set-up 

3.1 The interactive learning environment prototype 
Basic features 

An interactive learning environment is in our conception a demand driven environment in which 
students are actively involved in selecting, exploring and studying fiom a knowledge base of learning 
materials. Students can either browse through the learning materials in a sequential order by using the 
page-forward or page-backward button, or follow a 'hyperlink' strategy by choosing any part of the 
learning materials from the Topic option in the main menu. The basic content and the embedded support 
devices can be simultaneously presented in different windows. Choosing a topic (paragraph) returns by 
default the specific'basic content (cf. Figure 2). 



12.1 prior knowledge 
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If we v/ant to inv Advance organizer 
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Introduction 
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Prior knowledge 
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Global inactive 



Figure 2. 



Seleaion of ESD from the main menu. 



In the main meriubar students can click on the ESD option and 'select' any ESD. This selection represents 
only a global selection. However, the ESD will be available for further activation at the level of each 
topic/paragraph. At paragraph-level, the prototype helps to activate or deactivate pre-selected ESD. 
Activation of ESD is supported from within the local active window or from within the local inactive 
window (both in the right part of the screen). If an ESD is selected and active, it is 'embedded' in the 
basic content, as represented in the central text screen (Figure 3). 



Er|c i3 



UMmmmmm 



comparing learning environments 



loplc looU' .Support devtce> ^vwnf 

12.1 ~ 



If we want investigate a large population, it is impossible to 
include all subjects who belong to this population in our 
research. In such cases the research has to be restricted to a 
limited part of the population, a sample. Sample features are 
used as a means to derive statistically-based conclusions about 
the whole population. This way the results of a specific situation 
are generalized to a general situation. Wa call this type of 
research: inductive statistics. 

Example 

A car manufacturer claims that less than 2% of all produced 
cars have a malfunction. In order to investigate this claim It 
is not necessary tn include all produced cars In our research. 
The claim can also be tested by drawing a carefully selected 
sample of cars that Is representative for the whole 
production. The sample results enable us to estimate the 
percentage of cars that have a malfunction. 



'mm 



Global active ♦ 



Prior knowledge 
Introduction 



Global inactive 



Figure 3. Integration of tSD in basic content. 



Figure 3 clearly shows that ESD are separated from the basic content. Tables and illustrations can be 
accessed by clicking hot-words. 

The Tools option in the main menubar gives access to a history provision and the glossary. Activation 
of the /^isforv-tool lists all paragraphs that already have been looked at during a session. Activation of the 
second tool, the glossary, results in the presentation of an alphabetic list including all main concepts of 
the statistics domain covered in the set of learning materials. Selecting one of the concepts leads to the 
presentation of a short definition of that topic. Furthermore, the glossary can - on demand - provide the 
student with an example and if needed, the student can add an annotation to the definition. This 
annotation is stoi '^d separately for each individual student. 

The fourth option in the main menubar. Answers, gives access to the correct answers and solutions 
for the tasks and exercises included as ESD in the learning materials. 

For investigation purposes, all student interactions with the system are stored in a log-file. ESD- 
interactions, use of hyperlink features and glossary consultation are considered as measures for active 
involvement of a student. Note that discernability of ESD is one of the conditions and cannot be 
manipulated by students in this experiment. 

3.2 Versions of ILE 

The distinction between basic content and ESD is a basic feature in our ILE. Considering the degree of 
discernability of the ESD and the degree of inieractivily in selecting and/or activating the ESD, three 
versions of interactive learninj', environments have been elaborated. 

In a first version, the interaction possibilities with the ESD have been blocked. In this way the 
imeraciivc learning environment is reduced to a simple, straightforward electronic textbook. 

In two other versions, students are presented with a different way of embedding the support devices; 
the I-.sn are cither discernablc or not discernable in the context of the basic content. In the former 
selling;, llic l.SI> .uui b.viic (■onicnl ,'rc clearly sep.iratwi, and cirh ESD is preceded by a header, indicalin|; 

7 



comparing learning environments 



the type of ESD which is presented (cf. the example in Figure 3). In the latter setting the ESD and basic 
content are completely integrated. 

The versions of ILE are prototypes. Prototyping is seen as an essential part of courseware development to 
enhance the quality (cf. Gray and Black, 1994). 

3.3 Printed learning materials 
Basic features 

As stated earlier the Ou still mainly supports independent learning with printed learning materials. A 
variety of models has been elaborated; e.g., textbook-working book model, learning unit model, case 
model, etc. The printed learning environments have in common that the basic content is enhanced with a 
rich set of ESD. 

3.4 Versions of printed learning environments 

Comparable to the ILE the distinction between basic content and embedded support devices is a basic one. 
But the degree of discernability of the embedded support devices can vary. For research purposes, two 
different versions of printed learning materials have been elaborated. In a first version, the ESD are 
discernable and are preceded by a header. In a second version, all ESD are completely integrated into the 
basic content and, as a consequence, no longer discernable. 

3.5 Traditional face-to-face lectures 

The traditional face-to-face lectures are held for large groups of students, with one professor giving 
lectures on inferential statistics. This group is interpreted as the control group. 

3.6 Methodology 

The entire first-year student population, studying 'Psychology and Educational Sciences' at the University 
of Gent, Belgium (n = 502) participated in the study. Students were assigned, randomly, to either one of 
five experimental groups or to the control group (Table 1). Students in the control group were not 
subjected to any particular treatment. They followed the regular face-to-face lectures, given by their usual 
professor. In the results section, only the scores and behaviour of students who participated in all 
experimental sessions or corresponding lectures will be used for statistical analysis. Moreover, students 
with a higher educational background (n"'31) and students who take the course for the second time 
(n = 79) are excluded from statistical analysis. 

Table 1: First-year student population at the University of Gent, Faculty of Psychology and 

Educational Sciences, Course Statistics I (N"502). 





Experimental groups 

Discernable ESD Non-discernable ESD 


Control group 


ILE 


I (n=40) 


ma (n=20) 

mb (n='20) non-interactive 


V (n--342) 


PLE 


n (n-40) 


rV (n-40) 



Considering the theoretical model for this research, a large number of processes and variables have to be 
dealt with. In part, certain variables can be controlled for by the specific research design (e.g., random 
selection of students). Some variables have to be explicitly measured or evaluated by making use of the 
following instruments: 

- a reading comprehension test; 
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- a prior-knowledge-state test used as a pre-test; 

- a subject-oriented mastery test used as a post-test and to determine study outcome; 

- a three-part questionnaire: 

"■ part A includes questions about motivation, age, study habits, educational level and 

gender; 

* part B asks students to judge the degree of accessibility of the learning materials studied; 

* part C puts forward questions about the vise and appreciation of each specific embedded 
support device. 

an attitude measuring scale in relation to learning with computers; 
The research procedure is reported in chronological order: 

* Administration of the reading comprehension test, prior knowledge state test and Ou-questionnaire 
part A 

Assignment of students to experimental or control groups. We controlled for differences in prior 
knowledge. 

"■ Introduction session to learn how to use the ILE 
"■ Taking the course: the alternative approaches 

- Students in the experimental conditions study during three sessions of two hours each with the 
specific learning environment. After each session they fill in the Ou-questionnaire, part C. 

- Students in the control condition follow the regular lecture-format. Three lectures were given, 
each lecture focusing on one of the selected chapters on inferential statistics. 

"■ Administration of the subject-oriented mastery test and Ou-questionnaire part B 

Test administration was organised one week after the research sessions were finished. 
The final examination 

At the end of the academic year, students have to take a statistics examination as a part of their final 
assessment of the first year. The results of this regular examinations are considered as long-term 
study outcome effects and can be compared partly with the scores of the subject-mastery test. 



4 Results 

Table 2 presents an overview of mean post-test scores and of the students in the different treatment 
conditions. In the further analyses the following techniques have been applied: regression analysis, 
ANOVA, MANOVA and Profile analysis. The latter technique is an extension of MANOVA and is 
especially useful to unravel interaction effects between multi-level independent variables, treatment 
conditions and dependent variables. 



Table 2: Post test scores as a function of experimental condition 



condition 


I 

electronic, 

discernable, 

interactive 


n 

paper 
discernable 


nia 

electronic, 

non-discernable, 

interactive 


Illb 

electronic, 

non-discernable 

non-interactive 


rv 

paper, 

non-discernable 


V 

lectures 


mean score 


7.58 


7.69 


7.65 


6.38 


8.19 


7.97 


N 


24 


29 


20 


8 


26 


132 



4.1 Use of ESD in ILE 
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Analysing the data of the on-line registration of student-activities in the ILE shows that about 47% of all 
ESD are actually used by the students. 

4.2 Comparing the different conditions 

Analysis of variance (ANOVA) is used to judge the significance of differences in mean post-test scores 
^able 2). No significant differences are found between the research conditions. Consequently, none of 
the conditions leads to better study outcomes than any other condition F (5,233) ".541 (p = .745). 

A similar analysis is used to test whether a difference appears later on. Analysis of variance reveals 
no significant effects of experimental condition on mean final examination scores. 

4.3 Delivery mode and study outcome 

In this analysis a comparison is made between the groups I, lUa and lUb (ile) and the groups U and IV 
(PLE). This comparison, based on ANOVA, does not show significant differences in mean study outcome 
F(l,bO)-1.441 (p».236). 

4.4 Interactivity mode and study outcome 

In this ANOVA, the results of group I and Illa are combined (fully interactive conditions) and compared 
with the results of group Hlb where the interactive selection of ESD was obstructed. Analysis of variance 
reveals no significant main effects nor interactions of the independent variables on study outcome 
(F(l,26)'= 1.048 (p=.315). However, multiple classification analysis (MCA) produces that the mean 
deviations from the grand mean of study outcome are negative in case ESD cannot be manipulated 
(selected). The groups that could work with a full interactive version of the prototype achieve 
predominantly higher study outcomes than the grand mean. 

4.5 Discernable versus non-discernable ESD 

For this analysis the mean scores of group I & n are compared with the mean post-test scores of group 
Ilia, and IV. The purpose of this analysis is to investigate whether discernability of ESD is beneficial. The 
main effect of the analysis of variance reveals no significant result. Interactive condition: F(l,42) = .O07 
(p = .935); paper condition: F(l,53)-=.499 (p-..183). 

Multivariate analysis of variance reveals no significant main effects of discernability, nor use of 
several t>'pes of ESD on study outcome. However, if the two-way interactions are taken into 
consideration, the analysis uncovers a significant interaction between discernability and the use of 
processing ESD (F(l,42) = 5.66, p<.05) on study outcome. Additionally we find a trend interaction 
between discernability and use of testing ESD (F(l,41)-3.67, p".06). The features of both interactions are 
analogous: students who use much processing and testing ESD benefit from a condition (treatment) in 
which ESD are non-discernable, i.e., integrated into the basic content. In contrast, low ESD-users seem to 
benefit from a condition in which ESD can be clearly identified. 

4.6 Effects of treatment conditions on the use of ESD 

Multivariate analyses of variance reveals no significant main effects nor interactions of discernability and 
interactivity mode on the use of ESD. However, the use of ESD is significantly affected by delivery mode. 
In order to analyze the effects of delivery mode on ESD-use we ust; the data that were collected by means 
of questionnaire part C. Again, the ESD are clustered according to the dimensions orienting, processing 
and testing and entered in a profile analysis procedure (parallelism test). The analysis produces a 
significant interaction between delivery mode and ESD-use, showing that the computer groups use more 
testing E.SD, whereas the paper groups use more orienting and processing ESD (Wilks' X=.74, p<.001). 

4.7 The impact of prior knowledge 
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In order to determine the effects of the prior knowledge state (PKS) on the use of tSD, we apply a 
multivariate analysis of variance in which PKS is defined as an independent variable and the use of 
orienting, processing and testing BSD as dependent variables. The analysis reveals a significant interaction 
between PKS and ESD-use in this sense that PKS has an enhancing effect on ESD-use. Moreover, this effect 
is significantly more pronounced for processing BSD and testing BSD (Wilks' X=.88, p<.05). The 
interaction effect is supported by univariate F-tests, which reveal that PKS has a significant, positive 
impact on the use of processing BSD (F(l,50)"6.12, p<.05) and the use of testing BSD (F(l,50) = 3.99, 
p - .05), but not on using orienting BSD. 

In an additional analysis we tried to unravel the prior knowledge effects. To this end, the data 
representing the two prior knowledge dimensions (behaviour and knowledge) were entered in stepwise 
multiple regression analyses as independent variables. As dependent variables we entered the three types 
of BSD: orienting, processing and testing BSD. The results (table 3) show that the use of processing and 
testing BSD can be predicted from prior knowledge scores on the know and relation parameters 
(positively related to the use of processing BSD), and scores on the insight and structure parameters 
(positively related to the use of testing BSD), 



Table 3: Stepwise multiple regressions equations (PIN- .05) for use of BSD types as a function of 

prior knowledge profile 



dependent variable 


predicting variables 


equation 




orienting BSD 


behavioural parameters: 
know, insight, apply 


none of the variables reached PIN ■= 


.05 limit 


orienting BSD 


knowledge parameters: 
concept, relation, 
structure, method 


none of the variables reached PIN = 


.05 limit 


processing BSD 


behavioural parameters: 
know, insight, apply 


procesd = .09 + .14 know 




processing BSD 


knowledge parameters: 
concept, relation, 
structure, method 


procesd - .11 + .10 relation 




testing BSD 


behavioural parameters: 
know, insight, apply 


testesd = .17 + .16 * insight 




testing BSD 


knowledge parameters: 
concept, relation, 
structure, method 


testesd - .16 + .17 structure 





Does the impact of PKS on ESD-use also influence study outcome, depending on the learning 
environment used? To answer this question, we compare the study o utcome of students as a 
function of their PKS and type of learning environment. We discover that high PKS-students 
achieve higher study outcomes than low PKS-students. This effect is most pronounced in the 
condition in which ESD are integrated into the basic content (F(l,43) = 4.33, p<.05). 

4.8 The impact of motivation 

Profile analysis (parallelism test) of the effect of motivation on the use of the three ESD-types 
uncovers no significant main effect nor any significant interaction. In contrast, the analyses on 
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the effects of attitude towards learning with computers revealed a significant main effect (Wilks' 
X=.89, p<.05), reflecting that a positive attitude leads to an increased ESD-use. This effect was 
significantly more important for the use of testing ESD (Wilks' X=.91, p<.05). 

Analysis of variance revealed no significant main effects nor interactions between attitude 
towards learning with computers and discernability mode or interactivity mode. The picture 
that arises if we concentrate on delivery mode is completely different. The results indicate a 
classical disordinal interaction which is supported by a significant interaction effect between 
attitude and delivery mode on study outcome (F(l,93)=4.32, p<.05). The interaction implies 
that the choice of a certain delivery mode depends on a student's attitude towards learning with 
computers. Students who like learning with computers will achieve better study outcomes in a 
computer condition, but will achieve worse study outcomes if they have to study from printed 
learning materials. 

4.9 The impact of reading comprehension 

The results of the reading comprehension test are entered in regression analyses in order to 
investigate the predictive power towards the use of ESD. None of the analyses reveal significant 
effects, which means that reading comprehension level and use of ESD are seemingly unrelated. 
However, if we take Pearson's r correlations into account, these appear to be slightly negative 
for all types of ESD. This might indicate a weak tendency that a low reading comprehension 
level is compensated by using more ESD (reading comprehension and use of orienting ESD: r=- 
.039; p = .765; reading comprehension and use of processing ESD: r=-.025; p = .841; reading 
comprehension and use of testing ESD: r=-.03O; p = .551). 



5 Discussion 

The discussion and interpretation of the results, will be structured according to the hypotheses. 

Hypothesis 1: Students studying in ILE will achieve a comparable study outcome as students studying 
in alternative learning environments. 

Since the resul':s reveal no significant differences in study outcome when comparing students in 
the treatment conditions, the hypothesis can be accepted. 

For more information on the comparison of independent learning groups with the control 
group who received traditional learning approaches we refer to Dekeyser, Goeminne, Schuyten, 
Martens, Portier, Valcke & Van Buuren (in prep.). Our rf^sults confirm earlier findings of Weges 
and Ellerman (1994), and Portier and Van Buuren (in press). Both studies evidenced that 
students who study in a computer-based learning environment are able to pass the final 
examinations, achieve equal results if compared to students in a traditional educational setting. 
Doubts towards computer-based delivery as for instance expressed by Kozma (1991) can be 
rejected. Students are able to acquire the necessary knowledge and skills by reading and learning 
from a computer display. 

Hypothesis 2: Subjects who use ESD benefit (higher study outcome) from the undiscernible condition. 
In contrast, subjects who do not use ESD, benefit from learning materials where ESD are clearly 
discemable. 
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The results show a significant interaction: students who use many processing and testing ESD 
benefit from a condition in which ESD are integrated into the basic content of the learning 
materials. On the other hand, students who use only few ESD benefit from learning materials in 
which ESD are clearly discernable. The interaction was only significant for students studying in 
the ILE. As a result, this hypothesis can only partly be accepted. 

The results are in line with results reported by Portier and Van Buuren (in press): some 
students explicitly indicated that they would prefer a discernable delivery of ESD, whereas others 
seemed to prefer a completely integrated delivery. 

Hypothesis 3: Students who study in an interactive learning environment achieve a higher study 
outcome than students using a non-interactive learning environment. 

Considering the results, this hypothesis has to be rejected. Interactivity is one of the basic 
functionalities of the prototype of the interactive learning environment. At the theoretical level, 
interactivity was implemented in our approach because of a constructivist perspective on 
learning (e.g., Martin, 1984), which states that the acquisition of nev/ knowledge schemes 
benefits from an active self-directed learning process. 

A possible explanation for the non-significant differences can be found in the relatively 
small number of students who studied in the non-interactive learning environment setting. 

Hypothesis 4: The prior knowledge state of a student is positively related to the active use of the 
interactive learning environment, and is an interaction variable in the relation between treatment 
and study outcome. 

The analyses reveal that PKS is positively related to ESD-use. Moreover, it appears that this effect 
is especially present for processing and testing ESD. At a detailed level it is found that the use of 
processing ESD can be predicted from particular subscores of the PKS {know and relation). In 
parallel, the use of testing ESD is positively related to subscores on the insight and structure 
parameter. As a conclusion, we can state that the first part of the hypothesis can be accepted. 

If we research the interaction of prior knowledge between ESD-use and study outcome, we 
perceive that the effect is significantly more pronounced in a condition where ESD are not 
discernable, i.e. integrated into the basic content. Also the second part of the hypothesis can be 
accepted. This specific interaction can be explained as follows: Students with a high level of 
prior knowledge can compensate for the lack of structure in the low discernability condition (cf. 
Matthews, 1982); since they can rely on knowledge structures already available. Students with a 
lower level of prior knowledge do not have this knowledge, which leads to a negative impaa on 
their study outcome if the task environment is also unstruaured. 

Hypothesis 5: Good readers need less ESD to support their study activities. Reading comprehension is 
an interaction variable in the relation between treatment and study outcomes. 

Considering the non-significant analysis results, the first part of this hypothesis is rejected. 

In our theoretical model, the reading comprehension process is considered to be closely 
related to the learning process. Oostendorp and Peeck (1991) argue that reading does not always 
imply a learning intention directed at changing an existing cognitive struaure. The difference 
between reading and learning is associated with the intentionality of the reading process. Our 
reading comprehension test did focus on comprehending text fragments. However, the test did 
not focus on learning these text fragments. This might explain the weak relationship between 
reading comprehension level and the use of supportive elements in the learning materials. 

The analysis results also do not support the second part of the hypothesis, since there are 
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no significant interactions between reading comprehension level and any of the experimental 
conditions. The discussion about the 'intentionality' of the reading process is also relevant here. 
In addition, we may question what the relevance is of reading comprehension in the domain of 
learning statistics. It is possible that reading comprehension is clearly related to understanding 
and learning predominantly textual learning material, but a considerable part of a statistics 
course includes the application of methods (formulas), making exercises or solving problems. 

Hypothesis 6: Motivation and ESD-use are positively interrelated; their interaction influences scudy 
outcome. 

Our measures of motivation do not reveal an influence of motivation on the use of ESD-types. 
This is in contrast with our model, since we expected an influence of motivation on the use of 
ESD. 

The concept of attitude is related to the concept motivation. A questionnaire on attitude 
towards computers shows that students' attitude towards learning with computers appears to be 
positively related to ESD-use. The latter effect is significantly more pronounced for testing ESD. 
Post-hoc analysis showed a significant positive correlation between attitude and motivation. 

As to the interaction effect of motivation, influencing the use of the learning environment 
and as such influences post test scores, no significant interaction effects have been found. But if 
we introduce again the relation between 'motivation' and 'attitude towards learning with 
computers', a different picture appears. In the interactive learning environment, a students' 
attitude towards learning with computers is a significant, predictive variable for the variable 
study outcome. 

In summary, hypothesis 6 is to be rejected but via the relation with attitude towards computers 
there is a relevant influence of a non-cognitive variable. 

5.1 Conclusions 

Regarding the main aims of the present study, the research results show the relative effectivity 
of independent learning materials and of ILE. 

The ILE are not less effective than printed learning environments or face-to-face lectures. For 
efficiency reasons face-to-face lectures might be replaced by self study conditions. It must be 
noted however that the conditions in which we used self study materials where not completely 
self study conditions, because the materials could not be studied at home. 

The extra possibilities that ILE offer did not result in better study outcomes than the other 
self study conditions, based on printed learning materials. The latter are much easier to make. 
This means that the present study did not reveal extra benefits from interactive learning 
materials. Some possible explanations are: 

In every condition students knew they had to achieve the same study results. This might 
imply that less optimal conditions are compensated for by for instance studying with more 
effort. This is a general problem in this type of research (Martens et al, 1995,a). 

The ILE used in this experiment was still based on text and support devices intended for 

printed self study materials. 

Although comparison of the different types of ILE does not result in significant differences 
in learning outcome, this study shows that individual student characteristics play a role in 
making use of the learning environment and as such influence study outcome. Depending on 
the degree to which particular settings of the interactive learning envirormient (discernability, 
interactivity, ESD-use) are in line with student characteristics, the effectivity of ILE is different. 

Extrapolating these findings to the future design of ILE, this implies that students might be 
advised -automatically- about a preferred combination of settings (e.g., to select or deselect ESD, 
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to make them discernabie or not, to get a printed version next to the interactive versions). 
However control to adapt the learning environment to individual needs should always be in the 
hands of the individual student. In future prototypes of ILE we will therefore try to improve the 
adaptivity to student characteristics and students' preferences. This will also imply the 
development of new, interactive ESD. In this way, we hope to develop and implement an 
interactive learning environment that optimizes the task environment to the needs of the 
individual student. 
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